TEACHER MODEL

Weight-Inherited Distillation for Task-Agnostic BERT Compression (2024.findings-naacl)

Challenge:	e-commerce product search is a key component of product discovery and sales in ecommerce . high computational demands of large transformer models pose challenges for their deployment in real-time scenarios.
Approach:	They propose a framework for real-time semantic matching that leverages both soft labels from a teacher model and ground truth generated from pairwise query-product and query-query signals.
Outcome:	The proposed framework outperforms teacher models and state-of-the-art models on e-commerce datasets.

Copied to clipboard

Taiqiang Wu, Cheng Hou, Shanshan Lao, Jiayi Li, Ngai Wong, Zhe Zhao, Yujiu Yang

Query Distillation: BERT-based Distillation for Ensemble Ranking (2020.coling-industry)

Challenge:	Knowledge Distillation (KD) is a predominant approach for BERT compression.
Approach:	They propose a weight-inherited distillation method which directly transfers knowledge from the teacher to a compact student model by inheriting the weights.
Outcome:	The proposed method outperforms state-of-the-art KD-based methods on GLUE and SQUAD benchmarks.

Copied to clipboard

Wangshu Zhang, Junhong Liu, Zujie Wen, Yafang Wang, Gerard de Melo

Text2Sql: Pure Fine-Tuning and Pure Knowledge Distillation (2025.naacl-industry)

Challenge:	Recent years have witnessed substantial progress in the development of neural ranking networks, but an increasingly heavy computational burden due to growing numbers of parameters and the adoption of model ensembles.
Approach:	They propose a two-stage distillation method that allows a smaller student model to be trained while benefiting from the better performance of the teacher model.
Outcome:	The proposed method shows higher-quality rankings compared to the teacher model.

Copied to clipboard

Gao yu Zhu, Wei Shao, Xichou Zhu, Lei Yu, Jiafeng Guo, Xueqi Cheng

Scaling Parameter-Constrained Language Models with Quality Data (2024.emnlp-industry)

Challenge:	Text2Sql is a task that translates natural language questions and database schemas into SQL queries.
Approach:	They employ pure fine-tuning strategy to reduce redundancy by using only 53% of the baseline prompt length to fine- tune the model.
Outcome:	The model outperforms the baseline model by 8.2% and 8.6% in Test-suite accuracy (TS) and exact-set-match accuracy (EM) under the most refined Spider dev set of prompts, the model achieves 73.5% and 75.4%, respectively, approaching state-of-the-art (SOTA) levels.

Copied to clipboard

Ernie Chang, Matteo Paltenghi, Yang Li, Pin-Jie Lin, Changsheng Zhao, Patrick Huber, Zechun Liu, Rastislav Rabatin, Yangyang Shi, Vikas Chandra

KD-Boost: Boosting Real-Time Semantic Matching in E-commerce with Knowledge Distillation (2023.emnlp-industry)

Challenge:	Scaling laws in language modeling quantify training loss as a function of dataset size and model parameters, but neglect the critical role of data quality in model generalization.
Approach:	They propose to use effective training tokens as a combination of text diversity and syntheticity as measured by a teacher model to calculate scaling laws.
Outcome:	The proposed term effective training tokens is a combination of two readily-computed indicators of text diversity and syntheticity as measured by a teacher model.

Copied to clipboard

Sanjay Agrawal, Vivek Sembium, Ankith M S

Evaluating Explanations: How Much Do Explanations from the Teacher Aid Students? (2022.tacl-1)

Challenge:	Existing SOTA techniques for semantic matching are mostly based on Siamese networks.
Approach:	They propose a novel knowledge distillation algorithm designed for real-time semantic matching . they train low latency accurate student models by leveraging soft labels from a teacher model .
Outcome:	The proposed algorithm outperforms teacher and SOTA knowledge distillation benchmarks on e-commerce datasets.

Copied to clipboard

Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary C. Lipton, Graham Neubig, William W. Cohen

Explainability & Interpretability in NLP

Challenge:	Existing methods to explain predictions by highlighting salient features are often unstated.
Approach:	They propose a framework to quantify the value of explanations via the accuracy gains that they confer on a student model trained to simulate a teacher model.
Outcome:	The proposed framework allows principled, automatic, model-agnostic evaluation of attributions.

When Speed Meets Intelligence: Scalable Conversational NER in an Ever-evolving World (2026.eacl-industry)

Copied to clipboard

Karim Ghonim, Antonio Roberto, Davide Bernardi

X-Instruction: Aligning Language Model in Low-resource Languages with Self-curated Cross-lingual Instructions (2024.findings-acl)

Challenge:	Large Language Models excel at understanding conversational semantics, but lack of data makes them impractical for production deployment.
Approach:	They propose a pipeline for generating multilingual conversational NER datasets with minimal human validation and a framework that leverages LLMs as semantic filters combined with catalog-based entity grounding to label live traffic data.
Outcome:	The proposed framework outperforms existing models on public and private conversations by 97.12% on CoNLL-2003 and 83.09% on OntoNotes 5.0.

Copied to clipboard

Chong Li, Wen Yang, Jiajun Zhang, Jinliang Lu, Shaonan Wang, Chengqing Zong

MKT: A Multi-Stage Knowledge Transfer Framework to Mitigate Catastrophic Forgetting in Multi-Domain Chinese Spelling Correction (2025.emnlp-industry)

Challenge:	Large language models respond well in high-resource languages but struggle in low-resourced languages.
Approach:	They propose a method to construct cross-lingual instruction following samples with instruction in English and response in low-resource languages.
Outcome:	The proposed method builds a large-scale cross-lingual instruction tuning dataset on 10 languages.

Copied to clipboard

Peng Xing, Yinghui Li, Shirong Ma, Xinnian Liang, Haojing Huang, Yangning Li, Shu-Yu Guo, Hai-Tao Zheng, Wenhao Jiang, Ying Shen

Green & Sustainable NLP

Challenge:	Chinese Spelling Correction (CSC) is a model that detects and corrects spelling errors in given sentences.
Approach:	They propose a model-agnostic model with an evolving teacher model and dynamic distillation weights for knowledge transfer in each domain rather than focusing solely on new domain knowledge.
Outcome:	The proposed model-agnostic framework is based on an evolving teacher model and dynamic distillation weights for knowledge transfer in each domain, rather than focusing solely on new domain knowledge.

Reasoning Visual Dialog with Sparse Graph Learning and Knowledge Transfer (2021.findings-emnlp)

Copied to clipboard

Gi-Cheon Kang, Junseok Park, Hwaran Lee, Byoung-Tak Zhang, Jin-Hwa Kim

Dynamic Knowledge Distillation for Pre-trained Language Models (2021.emnlp-main)

Challenge:	Visual dialog is a task of answering questions grounded in an image using dialog history as context.
Approach:	They propose a Sparse Graph Learning method to formulate visual dialog as a graph structure learning task.
Outcome:	The proposed model outperforms the state-of-the-art models on the VisDial v1.0 dataset.

Copied to clipboard

Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, Xu Sun

C2DLM: Causal Concept-Guided Diffusion Large Language Models (2026.findings-acl)

Challenge:	Existing methods conduct knowledge distillation statically, e.g., student model aligns output distribution to teacher model on pre-defined training dataset.
Approach:	They propose a dynamic knowledge distillation that empowers the student to adjust the learning procedure according to its competency . they find it is promising and provide discussions on potential future directions towards more efficient methods .
Outcome:	The proposed method can boost student model performance while accelerating training . the proposed method reduces memory usage and accelerates model inference .

Copied to clipboard

Kairong Han, Nuanqiao Shan, Ziyu Zhao, Zijing Hu, Xinpeng Dong, Ye Jun Jian, Lujia Pan, Fei Wu, Kun Kuang

KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning (2024.naacl-long)

Challenge:	Autoregressive (AR) and diffusion language models (DLMs) suffer from insufficient reasoning capabilities.
Approach:	They propose a fully connected Diffusion Language Model that uses a concept-level causal graph to guide attention to learn causal relationships between concepts.
Outcome:	The proposed model achieves a 12% improvement and 3.2 training speedup on the COT-OrderPerturb task, along with an average gain of 1.31% across six downstream reasoning tasks.

Copied to clipboard

Cong-Duy Nguyen, Thong Nguyen, Xiaobao Wu, Anh Tuan Luu

Distilling Multilingual Transformers into CNNs for Scalable Intent Classification (2022.emnlp-industry)

Challenge:	Existing work on multimodal sentence embeddings took negative samples without reviewing, resulting in noisy and noisy negative samples.
Approach:	They propose a multimodal contrastive learning approach that inherits the knowledge from the teacher model to learn the difference between positive and negative instances.
Outcome:	The proposed approach can detect noisy and wrong negative samples before they are calculated in the contrastive objective.

Copied to clipboard

Besnik Fetahu, Akash Veeragouni, Oleg Rokhlenko, Shervin Malmasi

Conversational Question Answering with Language Models Generated Reformulations over Knowledge Graph (2024.findings-acl)

Challenge:	Existing multilingual models for voice assistants are limited by their prohibitive inference time and limited performance.
Approach:	They propose to distill and deploy multilingual Transformer models for voice assistants using a teacher-student framework that uses teacher-trained models to supervise student model training.
Outcome:	The proposed model outperforms a teacher model trained on unlabelled data and achieves equivalent performance.

Copied to clipboard

Lihui Liu, Blaine Hill, Boxin Du, Fei Wang, Hanghang Tong

Natural Language Interfaces

Challenge:	State-of-the-art ConvQA methods struggle with inexplicit question-answer pairs, which can degrade ConvQ performance.
Approach:	They propose a reinforcement learning based model, CoRnNet, which utilizes question reformulations generated by large language models to improve ConvQA performance.
Outcome:	The proposed model outperforms state-of-the-art ConvQA models by using question reformulations generated by large language models (LLMs).

Reducing cohort bias in natural language understanding systems with targeted self-training scheme (2023.acl-industry)

Copied to clipboard

Dieu-thu Le, Gabriela Hernandez, Bei Chen, Melanie Bradford

Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal (2022.findings-acl)

Challenge:	In deep learning models, it is hard to capture all the variations of the language that different users can use.
Approach:	They propose a framework that uses four active learning strategies to identify important samples coming from new users and a self training phase where a teacher model is trained from the first phase to expand the training data with relevant cohort utterances.
Outcome:	The proposed framework reduces the bias related to new customers in a digital voice assistant system by using two phases: a fixing phase and a self training phase.

Copied to clipboard

Umang Gupta, Jwala Dhamala, Varun Kumar, Apurv Verma, Yada Pruksachatkun, Satyapriya Krishna, Rahul Gupta, Kai-Wei Chang, Greg Ver Steeg, Aram Galstyan

Align-to-Distill: Trainable Attention Alignment for Knowledge Distillation in Neural Machine Translation (2024.lrec-main)

Challenge:	Language models excel at generating coherent text, but can be biased in multiple ways, including the unfounded association of male and female genders with gender-neutral professions.
Approach:	They propose to modify teacher probabilities and augment the training set to learn a fair model during knowledge distillation by modifying teacher probability and augmenting the training sets.
Outcome:	The proposed approach reduces gender disparity in open-ended text generated from the distilled and finetuned models with only a minor compromise in utility.

Copied to clipboard

Heegon Jin, Seonil Son, Jemin Park, Youngseok Kim, Hyungjong Noh, Yeonsoo Lee

Think Wider, Detect Sharper: Reinforced Reference Coverage for Document-Level Self-Contradiction Detection (2025.emnlp-main)

Challenge:	Existing knowledge distillation approaches to NMT often rely on heuristics when deciding which teacher layers to distill from.
Approach:	They propose an approach to align student attention heads with their teacher counterparts by heuristics to solve a feature mapping problem.
Outcome:	The proposed strategy shows gains of +3.61 and +0.63 BLEU points for WMT-2022 DeDsb and WMT-2014 EnDe compared to baselines.

Copied to clipboard

Yuhao Chen, Yuanjie Lyu, Shuochen Liu, Chao Zhang, Junhui Lv, Tong Xu

On Curriculum Learning for Commonsense Reasoning (2022.naacl-main)

Challenge:	Recent approaches to document-level contradiction detection (DSCD) only gain marginal improvement and often introduce inconsistencies across repeated responses.
Approach:	They propose a method that combines supervised fine-tuning and reinforcement learning to enhance document-level contradiction detection (DSCD) they propose to use a task-specific reward function to expand the model’s reasoning scope, boosting both accuracy and consistency.
Outcome:	The proposed method significantly boosts Llama 3.1-8B-Instruct’s accuracy from 38.5% to 51.1%, and consistency from 59.6% to76.2%.

Copied to clipboard

Adyasha Maharana, Mohit Bansal

BiLD: Bi-directional Logits Difference Loss for Large Language Model Distillation (2025.coling-main)

Challenge:	Recent research suggests that data order can have a significant impact on the performance of finetuned models for natural language understanding.
Approach:	They use paced curriculum learning to rank data and sample training mini-batches with increasing levels of difficulty during finetuning.
Outcome:	The proposed model improves performance for socialIQA, CosmosQA, CODAH, HellaSwag, WinoGrande in both tuning settings.

Copied to clipboard

Minchong Li, Feng Zhou, Xiaohui Song

Teacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Models (2024.findings-acl)

Challenge:	Knowledge distillation (KD) is a method for reducing model size while preserving performance.
Approach:	They propose a method to distill large language models at the logit level by transferring knowledge from a large teacher model to a smaller student model.
Outcome:	The proposed method outperforms supervised fine-tuning, vanilla KL loss and five other distillation methods on 13 datasets.

Copied to clipboard

Adian Liusie, Yassir Fathullah, Mark Gales

Robustness in NLP

Challenge:	Large Language Models (LLMs) have demonstrated impressive zero-shot capabilities and versatility in NLP tasks, however they often fail to maintain crucial invariances for specific tasks.
Approach:	They propose to distill the capabilities of a computationally intensive, debiased, teacher model into a more compact student model.
Outcome:	The proposed model outperforms larger, biased teacher models by reducing the invariances of the student model by incorporating encoders.

Legal Mathematical Reasoning with LLMs: Procedural Alignment through Two-Stage Reinforcement Learning (2025.findings-emnlp)

Copied to clipboard

Kepu Zhang, Guofu Xie, Weijie Yu, Mingyue Xu, Xu Tang, Yaxin Li, Jun Xu

Mixed Distillation Helps Smaller Language Models Reason Better (2024.findings-emnlp)

Challenge:	Existing legal mathematical reasoning models lack structured numerical reasoning . existing models perform poorly on LexNum, while LexPam improves both mathematical accuracy and legal coherence.
Approach:	They propose a legal mathematical reasoning benchmark LexNum and LexPam to address this problem . LexPam is a two-stage reinforcement learning framework for efficient legal reasoning training.
Outcome:	The proposed framework improves mathematical accuracy and legal coherence . it also improves legal cohesion and generalizes effectively across tasks and domains.

Copied to clipboard

Li Chenglin, Qianglong Chen, Liangyue Li, Caiyu Wang, Feng Tao, Yicheng Li, Zulong Chen, Yin Zhang

Visual Editing with LLM-based Tool Chaining: An Efficient Distillation Approach for Real-Time Applications (2024.emnlp-industry)

Challenge:	Recent large language models (LLMs) have demonstrated impressive multiple step-by-step reasoning capabilities in recent NLP reasoning tasks.
Approach:	They propose a mixed distillation framework that distills multiple step-by-step reasoning abilities into smaller language models (SLMs) they leverage LLMs to generate multiple step by step reasoning rationales by sampling automatically.
Outcome:	The proposed framework outperforms existing models on SVAMP, GSM8K and ASDIV, while a single model generated by MD exceeds the comprehensive performance of two individual CoT and PoT distilled models.

Copied to clipboard

Oren Sultan, Alexander Khasin, Guy Shiran, Asnat Greenstein-Messica, Dafna Shahaf

RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation (2022.findings-naacl)

Challenge:	Using natural language as an interaction medium for video editing can mitigate the complexity of video editing.
Approach:	They propose a method to fine-tune LLMs for invoking tools in real-time applications by interpreting user stylistic requests in natural language.
Outcome:	The proposed model matches the performance of the teacher model significantly, reducing costs and latency.

Copied to clipboard

Md Akmal Haidar, Nithin Anchuri, Mehdi Rezagholizadeh, Abbas Ghaddar, Philippe Langlais, Pascal Poupart

MGCL: Multi-Granularity Clue Learning for Emotion-Cause Pair Extraction via Cross-Grained Knowledge Distillation (2024.findings-emnlp)

Challenge:	Existing methods for intermediate layer knowledge distillation suffer from computational burdens and engineering efforts for setting up a proper layer mapping.
Approach:	They propose a method where intermediate layers from teacher and student models are randomly selected to be distilled into intermediate layers of student models.
Outcome:	The proposed method outperforms state-of-the-art intermediate layer knowledge distillation methods on GLUE tasks and out-of domain test sets.

Copied to clipboard

Yang Yu, Xin Lin, Changqun Li, Shizhou Huang, Liang He

Combining Curriculum Learning and Knowledge Distillation for Dialogue Generation (2021.findings-emnlp)

Challenge:	Traditional methods often rely on coarse-grained clause-level annotations, which overlook valuable fine-grain clues.
Approach:	They propose a method that captures fine-grained clues from a weakly-supervised perspective efficiently by using a teacher model to give sub-clause clues without needing fine-grain annotations.
Outcome:	The proposed method achieves state-of-the-art performance while offering improved interpretability.

Copied to clipboard

Qingqing Zhu, Xiuying Chen, Pengfei Wu, JunFei Liu, Dongyan Zhao

Tailoring Instructions to Student’s Learning Levels Boosts Knowledge Distillation (2023.acl-long)

Challenge:	Existing studies have shown that curriculum learning facilitates dialogue generation tasks while knowledge distillation can yield significant performance boosts for student models.
Approach:	They propose a combination of curriculum learning and knowledge distillation for dialogue generation models . they cluster training cases according to their complexity and employ an adversarial training strategy .
Outcome:	The proposed model improves compared with baselines.

Copied to clipboard

Yuxin Ren, Zihan Zhong, Xingjian Shi, Yi Zhu, Chun Yuan, Mu Li

A Novel Three-stage Framework for Few-shot Named Entity Recognition (2024.lrec-main)

Challenge:	Recent success of natural language processing (NLP) is driven by the adoption of large-scale pretrained language models.
Approach:	They propose a method to determine the impact of distillation influence on student generalization ability by prioritizing samples likely to enhance the student's generalization abilities.
Outcome:	The proposed method outperforms 10 common knowledge distillation baselines on 6 text classification tasks in the GLUE benchmark.

Copied to clipboard

Shengjie Ji, Fang Kong

Information Extraction & Text Mining

Challenge:	Existing methods for Named Entity Recognition (NER) rely on labeled data, but data scarcity is a major challenge.
Approach:	They propose a framework for Few-shot Named Entity Recognition that can learn from limited labeled data and generalize to new domains.
Outcome:	The proposed framework surpasses existing methods on several benchmarks.

Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation (2024.findings-acl)

Copied to clipboard

Joe Stacey, Marek Rei

LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation (2023.eacl-main)

Challenge:	Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some performance benefits.
Approach:	They propose to augment the distillation with generated unlabelled examples that match the target distribution and upsamples data points among the training set that are similar to the target.
Outcome:	The proposed method outperforms previous robustness solutions on the task of natural language inference (NLI) it also improves performance on OOD domains even beyond the target domain.

Copied to clipboard

Zhuoyuan Mao, Tetsuji Nakagawa

Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step (2023.acl-long)

Challenge:	Large-scale language-agnostic sentence embedding models suffer from inference speed and computation overhead.
Approach:	They propose to train a lightweight sentence embedding model to achieve this by incorporating knowledge from a teacher model.
Outcome:	The proposed model can build low-dimensional sentences for 109 languages with a thin-deep encoder.

Copied to clipboard

Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, Yejin Choi

Teaching Small Language Models to Reason (2023.acl-short)

Challenge:	Symbolic Chain-of-thought Distillation (SCoTD) is a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model.
Approach:	They propose a method to train a smaller student model on rationalizations from a larger teacher model.
Outcome:	The proposed method improves the performance of a student model in supervised and few-shot settings and especially for challenge sets.

Copied to clipboard

Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, Aliaksei Severyn

A Simple Concatenation can Effectively Improve Speech Translation (2023.acl-short)

Challenge:	Chain of thought prompting improves reasoning capabilities of large language models but only emerges in models with tens of billions of parameters.
Approach:	They propose to fine tune a student model on chain of thought outputs generated by a larger teacher model.
Outcome:	The proposed method improves task performance across arithmetic, commonsense and symbolic reasoning datasets.

Copied to clipboard

Linlin Zhang, Kai Fan, Boxing Chen, Luo Si

Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment (2025.coling-main)

Challenge:	Experimental results show that in our unified cross-modal ST model, models can effectively utilize the auxiliary information from speech and text.
Approach:	They propose a unified cross-modal ST method which concatenates speech and text as the input and builds a teacher that can utilize both cross-modities simultaneously.
Outcome:	The proposed method can effectively utilize the auxiliary information from speech and text, and achieve compelling results on MuST-C datasets.

Copied to clipboard

Tianyu Peng, Jiajun Zhang

SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning (2026.findings-eacl)

Challenge:	Existing knowledge distillation techniques for large language models are causing difficulties for student models to learn multi-modal probability distributions.
Approach:	They propose a ranking loss-based knowledge distillation method that encourages consistency of the ranking of peak predictions between teacher and student models.
Outcome:	The proposed method improves student models' ability to learn multi-modal distributions.

Copied to clipboard

Kaiwen Zhou, Ahmed Elgohary, A S M Iftekhar, Amin Saied

Redistributing Low-Frequency Words: Making the Most of Monolingual Data in Non-Autoregressive Translation (2022.acl-long)

Challenge:	Existing red-teaming frameworks do not cover all the risks associated with arbitrary black-box LLMs.
Approach:	They propose a generic red-teaming framework for arbitrary black-box LLM agents that iteratively constructs and refines model-based adversarial attacks based on the execution trajectories of former attempts.
Outcome:	The proposed model improves attack success rate by 100%, surpassing the 671B Deepseek-R1 model.

Copied to clipboard

Liang Ding, Longyue Wang, Shuming Shi, Dacheng Tao, Zhaopeng Tu

Who Taught You That? Tracing Teachers in Model Distillation (2025.findings-acl)

Challenge:	Knowledge distillation (KD) is the preliminary step for training non-autoregressive translation models, but it can lose important information for translating low-frequency words.
Approach:	They propose a knowledge distillation method which trains NAT student on external monolingual data with AT teacher trained on the original bilingual data.
Outcome:	Extensive experiments on eight WMT benchmarks show that monolingual KD outperforms the standard KD by improving low-frequency word translation without introducing any computational cost.

Copied to clipboard

Somin Wadhwa, Chantal Shaib, Silvio Amir, Byron C Wallace

STAD: Self-Training with Ambiguous Data for Low-Resource Relation Extraction (2022.coling-1)

Challenge:	Xu et al., 2006, show that model distillation can imbue efficient small language models with task-specific capabilities competitive with expensive teacher LLMs.
Approach:	They propose to distill outputs from a large teacher model to a small student model . they propose to use part-of-speech templates as higher-order linguistic features capable of capturing distinctive signals from teacher models that persist in distilled student outputs.
Outcome:	The proposed model distillation technique can imbue efficient small language models with task-specific capabilities competitive with (expensive) teacher LLMs.

Copied to clipboard

Junjie Yu, Xing Wang, Jiangjiang Zhao, Chunjie Yang, Wenliang Chen

Information Extraction & Text Mining

Challenge:	Existing approaches for low-resource relation extraction use only confident instances and uncertain instances.
Approach:	They propose a self-training approach for low-resource relation extraction using auto-annotated instances.
Outcome:	The proposed method improves on two widely used datasets with low-resource settings.

Pre-training Distillation for Large Language Models: A Design Space Exploration (2025.acl-long)

Copied to clipboard

Hao Peng, Xin Lv, Yushi Bai, Zijun Yao, Jiajie Zhang, Lei Hou, Juanzi Li

Lion: Adversarial Distillation of Proprietary Large Language Models (2023.emnlp-main)

Challenge:	Knowledge distillation (KD) aims to transfer knowledge from a large teacher model to a smaller student model for model compression.
Approach:	They extend knowledge distillation to the pre-training phase of large language models . they first conduct an experiment using a teacher LLM to distill a 1.9B student LLM .
Outcome:	The proposed model can be used to distill a 1.9B student model using a teacher LLM.

Copied to clipboard

Yuxin Jiang, Chunkit Chan, Mingyang Chen, Wei Wang

Online Distilling from Checkpoints for Neural Machine Translation (N19-1)

Challenge:	Existing methods for transferring knowledge from a proprietary large language model to an open-source LLM have been criticized for not incorporating any feedback to boost student model's proficiency iteratively.
Approach:	They propose a novel adversarial distillation framework that integrates feedback to identify challenging instructions where the student model’s performance falls short.
Outcome:	The proposed framework surpasses existing state-of-the-art LLMs by 55.4% in challenging zero-shot reasoning benchmarks and 16.7% on AGIEval.

Copied to clipboard

Hao-Ran Wei, Shujian Huang, Ran Wang, Xin-yu Dai, Jiajun Chen

Query Optimization for Parametric Knowledge Refinement in Retrieval-Augmented Large Language Models (2025.findings-emnlp)

Challenge:	Existing neural machine translation models have a deep structure with large amounts of parameters, making them hard to train.
Approach:	They propose an online method to generate a teacher model from checkpoints . they show steady improvement over a strong self-attention-based baseline system .
Outcome:	The proposed method improves on-the-fly on several datasets and language pairs.

Copied to clipboard

Youan Cong, Pritom Saha Akash, Cheng Wang, Kevin Chen-Chuan Chang

MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (2020.acl-main)

Challenge:	Extract-Refine-Retrieve-Read is a query optimization framework for large language models . it is designed to bridge the pre-retrieval information gap in Retriev-Augmented Generation systems .
Approach:	They propose a framework to extract parametric knowledge from Large Language Models and refine them using a specialized query optimizer.
Outcome:	The extract-refine-retrieve-read framework outperforms baselines on QA datasets . it is designed to meet the knowledge requirements of large language models (LLMs)

Copied to clipboard

Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, Denny Zhou

DRP: Distilled Reasoning Pruning with Mathematical Skill-aware Step Decomposition for Efficient Large Reasoning Models (2026.findings-acl)

Challenge:	Empirical studies show that MobileBERT is 4.3x smaller and 5.5x faster than BERT_BASE . BERT is one of the largest models ever in NLP, but suffers from heavy model size and high latency .
Approach:	They propose a tool to compress and accelerate the popular BERT model by task-agnostic application.
Outcome:	The proposed model is 4.3x smaller and 5.5x faster than BERT_BASE . it achieves competitive results on well-known benchmarks .

Copied to clipboard

Yuxuan Jiang, Dawei Li, Francis Ferraro

Green & Sustainable NLP

Challenge:	Existing solutions to this problem are inference-time pruning and tuning-based distillation.
Approach:	They propose a framework that combines inference-time pruning with tuning-based distillation to enable efficient and accurate reasoning.
Outcome:	The proposed framework reduces token usage while improving accuracy on GSM8K and AIME tokens while avoiding performance drop.

GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment (2024.findings-acl)

Copied to clipboard

Yao Yao, Zuchao Li, Hai Zhao

SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models (2025.findings-naacl)

Challenge:	Existing methods of acceleration require fine-tuning of considerably large models, such as Llama-7B, posing a challenge for average users.
Approach:	They propose a Guidance-based Knowledge Transfer framework that leverages a larger LLM as a 'teacher' and a smaller 'student' model to finalize responses.
Outcome:	The proposed framework achieves a maximum accuracy improvement of 14.18%, along with a 10.72 times speed-up on GSM8K and an accuracy improvement 14.00% along with 7.73 times speed up in CSQA.

Copied to clipboard

Jahyun Koo, Yerin Hwang, Yongil Kim, Taegwan Kang, Hyunkyung Bae, Kyomin Jung

Annealing Knowledge Distillation (2021.eacl-main)

Challenge:	Knowledge Distillation (KD) has emerged as a popular method for compressing large language models due to high inference costs and memory requirements.
Approach:	They propose a method that integrates the teacher model during the student's sequence generation to reduce misguidance from the teacher.
Outcome:	Experiments on three model families and five instruction-following datasets show that SWITCH surpasses traditional methods, especially in the generation of long sequential data.

Copied to clipboard

Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma, Ali Ghodsi

Lifelong Language Knowledge Distillation (2020.emnlp-main)

Challenge:	Knowledge distillation (KD) is a powerful model compression technique for deep neural networks.
Approach:	They propose a method to feed the rich information provided by teacher’s soft-targets incrementally and more efficiently by annealing the teacher output incrementally.
Outcome:	The proposed method can be used on image classification and NLP language inference tasks with BERT-based models on the GLUE benchmark.

Copied to clipboard

Yung-Sung Chuang, Shang-Yu Su, Yun-Nung Chen

Recall with Reasoning: Chain-of-Thought Distillation for Mamba’s Long-Context Memory and Extrapolation (2025.emnlp-main)

Challenge:	Existing methods to perform lifelong language learning (LLL) on stream of different tasks are challenging . Existing models face catastrophic forgetting problem, which can be mitigated by lifelong learning .
Approach:	They propose a method that can be easily applied to existing LLL architectures to mitigate degradation.
Outcome:	The proposed method improves state-of-the-art models and reduces degradation compared to multi-task models.

Copied to clipboard

Jun-Yu Ma, Tianqing Fang, Zhisong Zhang, Hongming Zhang, Haitao Mi, Dong Yu

Text Generation

Challenge:	Existing long-context memory methods such as Mamba struggle with long-constituency when the length of the processed text exceeds the model's training length.
Approach:	They propose a method that uses chain-of-thought summarization to teach Mamba to actively recall and reason over long contexts.
Outcome:	Experiments on LONGMEMEVAL and HELMET show that RwR outperforms existing long-term memory methods while preserving short-context capabilities.

PILOT: Planning via Internalized Latent Optimization Trajectories for Large Language Models (2026.acl-long)

Copied to clipboard

Haoyu Zheng, Yun Zhu, Yuqian Yuan, Bo Yuan, Wenqiao Zhang, Siliang Tang, Jun Xiao

Quantification of Large Language Model Distillation (2025.acl-long)

Challenge:	Large Language Models lack the capacity to formulate global strategies due to latency and availability constraints.
Approach:	They propose a framework to internalize the strategic oversight of large models into intrinsic Latent Guidance by synthesizing a query-conditioned Latent Guide.
Outcome:	The proposed framework outperforms strong baselines on mathematical and coding benchmarks with negligible inference latency.

Copied to clipboard

Sunbowen Lee, Junting Zhou, Chang Ao, Kaige Li, Xeron Du, Sirui He, Haihong Wu, Tianci Liu, Jiaheng Liu, Hamid Alinejad-Rokny, Min Yang, Yitao Liang, Zhoufutu Wen, Shiwen Ni

ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation (2020.acl-main)

Challenge:	Existing studies have revealed the robustness degra-dation caused by data distillation.
Approach:	They propose a framework to evaluate and quantify model distillation . they aim to identify identity cognition contradictions and analyse multi-granularity response similarities across models to measure the extent of homogenization.
Outcome:	The proposed framework addresses two key aspects: (1) Identifying identity cognition contradictions to assess discrepancies in how models perceive and represent identity-related information; (2) Analyzing multi-granularity response similarities across models to measure the extent of homogenization.

Copied to clipboard

Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, Kevin Gimpel

Machine Translation

Challenge:	a non-autoregressive machine translation model can minimize the autoregressive teacher's energy . engINE is an inference network trained to minimize the teacher' energy based on distilled corpora .
Approach:	They propose to train a non-autoregressive machine translation model to minimize autoregressive teacher energy by using an inference network instead of distilled corpora.
Outcome:	The proposed model achieves state-of-the-art non-autoregressive results on two datasets . the proposed model is trained to minimize the autoregressive teacher energy .

Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillation (2025.findings-acl)

Copied to clipboard

Shuai Zhao, Xiaobao Wu, Cong-Duy T Nguyen, Yanhao Jia, Meihuizi Jia, Feng Yichao, Anh Tuan Luu

A Self-Distillation Recipe for Neural Machine Translation (2025.findings-acl)

Challenge:	Parameter-efficient fine-tuning (PEFT) can bridge the gap between large language models and downstream tasks, but is vulnerable to malicious attacks.
Approach:	They propose a weak-to-strong unlearning algorithm based on feature alignment knowledge distillation to defend against backdoor attacks . they first train a small-scale language model through full-parameter fine-tuning to serve as the clean teacher model and then guide the large-scale poisoned student model in unlearning the backdoor.
Outcome:	The proposed method can unlearn backdoor features without compromising model performance.

Copied to clipboard

Hongfei Xu, Zhuofei Liang, Qiuhui Liu, Lingling Mu

ELAD: Explanation-Guided Large Language Models Active Distillation (2024.findings-acl)

Challenge:	Existing methods for Neural Machine Translation (NMT) have been proven effective in improving the performance of computer vision tasks without pre-training a teacher.
Approach:	They propose a rank-order augmented Pearson correlation loss and an iterative distillation method to prevent the discrepancy of predictions between the student and a stronger teacher from disturbing the training.
Outcome:	The proposed method can lead to significant improvements over the strong Transformer baseline on low/middle/high-resource tasks, obtaining comparable or better performance with fewer layers.

Copied to clipboard

Yifei Zhang, Bo Pan, Chen Ling, Yuntong Hu, Liang Zhao

Dialogue Distillation: Open-Domain Dialogue Augmentation Using Unpaired Data (2020.emnlp-main)

Challenge:	Large Language Models (LLMs) are hindered by their memory inefficiency, computational demands, and the high costs of API inferences.
Approach:	They propose an Explanation-Guided LLMs Active Distillation framework that employs an active learning strategy to optimize the balance between annotation costs and model performance.
Outcome:	The proposed framework significantly improves the efficiency of LLMs knowledge distillation.

Copied to clipboard

Rongsheng Zhang, Yinhe Zheng, Jianzhi Shao, Xiaoxi Mao, Yadong Xi, Minlie Huang

NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application (2021.findings-emnlp)

Challenge:	Existing research has focused on training open-domain dialogue models using unpaired data.
Approach:	They propose a data-level distillation method for training open-domain dialogue models by utilizing unpaired data.
Outcome:	The proposed method produces high-quality dialogue pairs with diverse contents, and can improve competitive baselines.

Copied to clipboard

Chuhan Wu, Fangzhao Wu, Yang Yu, Tao Qi, Yongfeng Huang, Qi Liu

LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression (2020.coling-main)

Challenge:	Existing language models are pre-trained and distilled on general corpus like Wikipedia, which has gaps with the news domain and may be suboptimal for news intelligence.
Approach:	They propose a method to distill existing language models on Wikipedia to enable efficient news intelligence.
Outcome:	The proposed model can be used to build and test a news intelligence application on Wikipedia and Wikipedia.

Copied to clipboard

Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Quanlu Zhang, Yaming Yang, Yunhai Tong, Jing Bai

Class-Incremental Few-Shot Event Detection (2024.lrec-main)

Challenge:	Existing models that use knowledge distillation are memory-intensive and latency-prohibitive . Existing solutions that use this knowledge distilling framework are expensive .
Approach:	They propose a solution that uses weight pruning, matrix factorization and knowledge distillation to learn a smaller model.
Outcome:	The proposed model reduces the training overheads by an order of magnitude on public datasets while preserving state-of-the-art accuracy.

Copied to clipboard

Kailin Zhao, Xiaolong Jin, Long Bai, Jiafeng Guo, Xueqi Cheng

Hybrid Knowledge Transfer for Improved Cross-Lingual Event Detection via Hierarchical Sample Selection (2023.acl-long)

Challenge:	Existing methods to deal with new class of events with only a few labeled instances are challenging . old knowledge forgetting and new class overfitting are two problems in this task.
Approach:	They propose a task called class-incremental few-shot event detection to solve old knowledge forgetting and new class overfitting problems.
Outcome:	The proposed method reduces old knowledge forgetting and new class overfitting problems on two benchmark datasets.

Copied to clipboard

Luis Guzman Nateras, Franck Dernoncourt, Thien Nguyen

uDistil-Whisper: Label-Free Data Filtering for Knowledge Distillation in Low-Data Regimes (2025.naacl-long)

Challenge:	Recent efforts to train cross-lingual models on source language fail to take advantage of data transfer . current methods focus on learning task-specific information from syntactical features or word-label relations in target language.
Approach:	They propose a hybrid knowledge-transfer approach that leverages a teacher-student framework . the model is evaluated on a distinct target language for which there is no labeled data .
Outcome:	The proposed model achieves state-of-the-art results on 9 morphologically-diverse target languages across 3 distinct datasets.

Copied to clipboard

Abdul Waheed, Karima Kadaoui, Bhiksha Raj, Muhammad Abdul-Mageed

SCOTT: Self-Consistent Chain-of-Thought Distillation (2023.acl-long)

Challenge:	Recent work on distilling Whisper’s knowledge into small models using pseudo-labels shows promising performance while reducing the size by up to 50%.
Approach:	They propose a framework that distills Whisper’s knowledge into small models using pseudo-labels and reduces the size by up to 50%.
Outcome:	The proposed model outperforms the teacher model by 5-7 WER points and is 25-50% more efficient when scaling the data.

Copied to clipboard

Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, Xiang Ren

Collective Relevance Labeling for Passage Retrieval (2022.naacl-main)

Challenge:	Large language models (LMs) generate free-text rationales for their predictions via chain-of-thought prompting, but there is little guarantee that the generated rationale is consistent with LM’s predictions or faithfully justify the decisions.
Approach:	They propose a faithful knowledge distillation method to learn a small, self-consistent CoT model from a larger teacher model by contrastive decoding.
Outcome:	The proposed method yields comparable performance but is less faithful than baselines.

Copied to clipboard

Jihyuk Kim, Minsoo Kim, Seung-won Hwang

Information Retrieval

Challenge:	Existing approaches to improve IR labels are incomplete and require computational overheads.
Approach:	They propose to distill knowledge for informed labeling without high computation overheads at evaluation time.
Outcome:	The proposed model outperforms state-of-the-art models while distilling the rankings better.

Causal Distillation for Language Models (2022.naacl-main)

Copied to clipboard

Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, Noah Goodman

Mutual-Learning Improves End-to-End Speech Translation (2021.emnlp-main)

Challenge:	Distillation efforts have led to language models that are more compact and efficient without serious drops in performance.
Approach:	They propose to augment distillation with a third objective that encourages the student model to imitate the causal dynamics of the teacher through a distillation interchange intervention training objective (DIITO).
Outcome:	The proposed method lowers perplexity on the WikiText-103M corpus and improves on the GLUE benchmark, SQuAD, and CoNLL-2003.

Copied to clipboard

Jiawei Zhao, Wei Luo, Boxing Chen, Andrew Gilman

Speech & Audio in NLP

Challenge:	Existing approaches to end-to-end speech translation (E2E) models only allow one way knowledge transfer, which is limited by the performance of the teacher model.
Approach:	They propose a one-way knowledge transfer paradigm where the MT and ST models are collaboratively trained and considered as peers rather than teacher/student.
Outcome:	The proposed model improves the performance of end-to-end speech translation (ST) task by combining knowledge from two models with peer models.

Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2022.naacl-main)

Copied to clipboard

Peter West, Chandra Bhagavatula, Jack Hessel, Jena Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, Yejin Choi

Wider & Closer: Mixture of Short-channel Distillers for Zero-shot Cross-lingual Named Entity Recognition (2022.emnlp-main)

Challenge:	Prior studies suggested pre-trained language models possess limited understanding of commonsense knowledge despite otherwise stellar performance on leaderboards.
Approach:	They propose a framework that uses larger models to teach smaller models by distilling knowledge symbolically as text in addition to the neural model.
Outcome:	The proposed framework is based on a general language model teacher's commonsense knowledge graphs and a neural commonsensing model surpassing the teacher model's in all three criteria.

Copied to clipboard

Jun-Yu Ma, Beiduo Chen, Jia-Chen Gu, Zhenhua Ling, Wu Guo, Quan Liu, Zhigang Chen, Cong Liu

Rethinking Semi-supervised Learning with Language Models (2023.findings-acl)

Challenge:	Existing mainstream methods for zero-shot cross-lingual named entity recognition ignore the rich and complementary information lying in the intermediate layers of pre-trained language models and domain-invariant information is easily lost during transfer.
Approach:	They propose a mixture of short-channel distillers to fully interact the rich hierarchical information in the teacher model and to transfer knowledge to the student model sufficiently and efficiently.
Outcome:	The proposed method shows great generalization and compatibility across languages and fields.

Copied to clipboard

Zhengxiang Shi, Francesco Tonolini, Nikolaos Aletras, Emine Yilmaz, Gabriella Kazai, Yunlong Jiao

Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging (2022.findings-emnlp)

Challenge:	Semi-supervised learning (SSL) is a popular setting to make use of unlabelled data . Currently, there are two popular approaches to make effective use of the unlabelled datasets .
Approach:	They compare semi-supervised learning (SSL) and task-adaptive pre-training (TAPT) they find TAPT is a stronger and more robust SSL learner, even when using just a few hundred unlabelled samples .
Outcome:	The proposed methods improve model performance across different NLP tasks and data sizes.

Copied to clipboard

Peng Lu, Ivan Kobyzev, Mehdi Rezagholizadeh, Ahmad Rashid, Ali Ghodsi, Phillippe Langlais

PromptKD: Distilling Student-Friendly Knowledge for Generative Language Models via Prompt Tuning (2024.findings-emnlp)

Challenge:	Recent studies show that the flatness of the local minimum correlates well with better generalization.
Approach:	They propose to use a method encouraging convergence to a flatter minimum to fine-tune PLMs.
Outcome:	The proposed method outperforms state-of-the-art methods on NLP tasks without extra computation cost.

Copied to clipboard

Gyeongman Kim, Doohyuk Jang, Eunho Yang

Learn What Is Possible, Then Choose What Is Best: Disentangling One-To-Many Relations in Language Through Text-based Games (2022.findings-emnlp)

Challenge:	Recent advances in large language models (LLMs) have raised concerns about inference costs, increasing the need for research into model compression.
Approach:	They propose a method that utilizes prompt tuning to enable generative language models to transfer student-friendly knowledge.
Outcome:	Extensive experiments on instruction-following datasets show that PromptKD achieves state-of-the-art performance while adding only 0.0007% of the teacher’s parameters as prompts.

Copied to clipboard

Benjamin Towle, Ke Zhou

EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning (2024.emnlp-main)

Challenge:	Language models pre-trained on large self-supervised corpora, followed by task-specific fine-tuning has become the dominant paradigm in NLP.
Approach:	They propose to train language models pre-trained on large self-supervised corpora, followed by task-specific fine-tuning on the target domain.
Outcome:	The proposed model improves on the previous state-of-the-art model on the Jericho Walkthroughs dataset by 49%.

Copied to clipboard

Ashish Seth, Ramaneswaran Selvakumar, S Sakshi, Sonal Kumar, Sreyan Ghosh, Dinesh Manocha

Enhancing Chinese Word Segmentation via Pseudo Labels for Practicability (2021.findings-acl)

Challenge:	EH-MAM is a self-supervised learning approach for speech representation learning . prior methods used random masking schemes to learn speech representations .
Approach:	They propose a self-supervised approach that automatically selects hard regions during SSL training and introduces them to the model for reconstruction.
Outcome:	The proposed approach outperforms state-of-the-art models across low-resource speech recognition and SUPERB benchmarks by 5%-10%.

Copied to clipboard

Kaiyu Huang, Junpeng Liu, Degen Huang, Deyi Xiong, Zhuang Liu, Jinsong Su

Syntactic Text Processing

Challenge:	Pre-trained language models alleviate segmentation ambiguity and out-of-vocabulary (OOV) words.
Approach:	They propose a semisupervised neural method which distills knowledge from unlabeled data to a student model to improve both in-domain and out-of-domain CWS.
Outcome:	The proposed method can keep practicability of the lightweight student model and improve segmentation effectively on downstream Chinese NLP tasks.

Dimension Reduction for Efficient Dense Retrieval via Conditional Autoencoder (2022.emnlp-main)

Copied to clipboard

Zhenghao Liu, Han Zhang, Chenyan Xiong, Zhiyuan Liu, Yu Gu, Xiaohua Li

Self-Training using Rules of Grammar for Few-Shot NLU (2021.findings-emnlp)

Challenge:	Existing work reserves the principle dimensions of query and document embeddings for building more efficient retrieval systems.
Approach:	They propose to use Conditional Autoencoder to compress high-dimensional embeddings to maintain the same embeddable distribution and better recover ranking features.
Outcome:	The proposed algorithm achieves comparable ranking performance with its teacher model and makes the retrieval system more efficient.

Copied to clipboard

Joonghyuk Hahn, Hyunjoon Cheon, Kyuyeol Han, Cheongjae Lee, Junseok Kim, Yo-Sub Han

StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion (2024.acl-long)

Challenge:	Existing methods for learning natural language understanding are limited in low-resource settings.
Approach:	They propose to use rules of grammar to construct and expand rules of grammatical structure of data without human involvement.
Outcome:	The proposed approach outperforms state-of-the-art methods in three benchmark datasets.

Copied to clipboard

Zhichao Wang, Yuanzhe Chen, Xinsheng Wang, Lei Xie, Yuping Wang

QBridge: Bridging Natural Language and SQL via Gold Query Rewriting with Agentic Refinement (2026.acl-long)

Challenge:	Existing LM-based VC models require offline conversion from source semantics to acoustic features, limiting their deployment to real-time applications.
Approach:	They propose a streaming LM-based model for zero-shot voice conversion that uses a fully causal context-aware LM with a temporal-independent acoustic predictor to facilitate real-time conversion given arbitrary speaker prompts and source speech.
Outcome:	The proposed model achieves comparable performance to non-streaming VC systems while maintaining a fully causal context-aware LM with a temporal-independent acoustic predictor.

Copied to clipboard

Zhensheng Luo, Sai Wu, Yuan Qiu, Chang Yao, Gang Chen, Xiu Tang

Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text (2023.acl-long)

Challenge:	Natural language to SQL (NL2SQL) is an intuitive interface for querying structured data . but real user questions are noisy, ambiguous, and weakly grounded to database semantics.
Approach:	They propose an agentic feedback-driven NL2SQL framework that bridges natural language and SQL via Gold Query.
Outcome:	The proposed framework outperforms strong prompting and agentic baselines on spider, BIRD, and three robustness variants on NL2SQL.

Copied to clipboard

Qianhui Wu, Huiqiang Jiang, Haonan Yin, Börje Karlsson, Chin-Yew Lin

Learning from Imperfect Data: Towards Efficient Knowledge Distillation of Autoregressive Language Models for Text-to-SQL (2024.findings-emnlp)

Challenge:	Self-supervised representation learning has proved to be a valuable component for out-of-distribution (OoD) detection with only the texts of in-difference (ID) examples.
Approach:	They propose a method that integrates strengths and weaknesses of both methods . they use a fine-tuned model as the teacher to teach a randomly initialized student model .
Outcome:	The proposed method outperforms human evaluators in the pair-expert task on the Human ChatGPT Comparison Corpus.

Copied to clipboard

Qihuang Zhong, Kunfeng Chen, Liang Ding, Juhua Liu, Bo Du, Dacheng Tao

SKD-NER: Continual Named Entity Recognition via Span-based Knowledge Distillation with Reinforcement Learning (2023.emnlp-main)

Challenge:	Existing text-to-SQL LLMs are computationally expensive and difficult to deploy in real-world applications.
Approach:	They propose to distill a larger teacher model into a smaller student model by using imperfect data to improve the KD.
Outcome:	The proposed method achieves the best tradeoff between performance and efficiency on 5 text-to-SQL benchmarks.

Copied to clipboard

Yi Chen, Liang He

Intermediate Layer Distillation with the Reused Teacher Classifier: A Study on the Importance of the Classifier of Attention-based Models (2024.findings-emnlp)

Challenge:	Continual learning for named entity recognition (CL-NER) aims to enable models to continuously learn new entity types while retaining the ability to recognize previously learned ones.
Approach:	They propose a model that leverages knowledge distillation to retain memory and employs reinforcement learning strategies to optimize the soft labeling and distillation losses generated by the teacher model to effectively prevent catastrophic forgetting.
Outcome:	The proposed model outperforms state-of-the-art models on two benchmark datasets showing that it significantly improves the performance of the CL-NER task.

Copied to clipboard

Hang Zhang, Seyyed Mozafari, James Clark, Brett Meyer, Warren Gross

Reverse Thinking Makes LLMs Stronger Reasoners (2025.naacl-long)

Challenge:	Existing methods underestimate the importance of utilizing the teacher's discriminative classifier and face challenges in establishing proper layer mappings.
Approach:	They propose to reuse pre-trained teacher classifiers to improve student performance . they use projectors to match hidden size of the teacher model to student .
Outcome:	The proposed method outperforms existing methods on 97.7% of the teacher BERT base without additional trainable parameters.

Copied to clipboard

Justin Chen, Zifeng Wang, Hamid Palangi, Rujun Han, Sayna Ebrahimi, Long Le, Vincent Perot, Swaroop Mishra, Mohit Bansal, Chen-Yu Lee, Tomas Pfister

Patient Knowledge Distillation for BERT Model Compression (D19-1)

Challenge:	Reverse-Enhanced Thinking (RevThink) is a framework for large language models to perform reverse thinking.
Approach:	They propose a framework for enhancing forward-backward reasoning by collecting data from a teacher model and employing three objectives to train a student model in a multi-task learning fashion.
Outcome:	The proposed framework outperforms a fine-tuning method trained on 10x more forward reasoning on 12 datasets covering commonsense, math, and logical reasoning.

Copied to clipboard

Siqi Sun, Yu Cheng, Zhe Gan, Jingjing Liu

Multi-Grained Knowledge Distillation for Named Entity Recognition (2021.naacl-main)

Challenge:	Pre-trained language models such as BERT have proven to be highly effective for natural language processing tasks, but the high demand for computing resources hinders their application in practice.
Approach:	They propose to compress an original large model (teacher) into an equally-effective lightweight shallow network (student) Empirically, this translates into improved results on multiple NLP tasks with a significant gain in training efficiency, without sacrificing model accuracy.
Outcome:	The proposed model reduces the computational cost of training models using the teacher model into a lightweight shallow network.

Copied to clipboard

Xuan Zhou, Xiao Zhang, Chenyang Tao, Junya Chen, Bing Xu, Wei Wang, Jing Xiao

Green & Sustainable NLP

Challenge:	Pre-trained big models have delivered top performance in Seq2seq modeling, but their deployments in real-world applications are often hindered by excessive computations and memory demands.
Approach:	They propose a distillation scheme to efficiently transfer knowledge from big models to their cheaper counterparts.
Outcome:	The proposed scheme maximizes the assimilation of knowledge from the teacher model to the student model.

Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection (2022.findings-emnlp)

Copied to clipboard

Chenglong Wang, Yi Lu, Yongyu Mu, Yimin Hu, Tong Xiao, Jingbo Zhu

Retrieved In-Context Principles from Previous Mistakes (2024.emnlp-main)

Challenge:	Existing studies on knowledge distillation have shown that not all knowledge is necessary for learning a good student model.
Approach:	They propose an actor-critic approach to selecting appropriate knowledge to transfer during the process of knowledge distillation.
Outcome:	The proposed method outperforms several strong knowledge distillation baselines significantly on the GLUE datasets.

Copied to clipboard

Hao Sun, Yong Jiang, Bo Wang, Yingyan Hou, Yan Zhang, Pengjun Xie, Fei Huang

Low-Resource NLP

Challenge:	Recent advances in in-context learning (ICL) have limited customization and inadequate error coverage.
Approach:	They propose a method to retrieve in-context principles from mistakes to improve model performance.
Outcome:	The proposed framework enhances model performance when applied to various prompting strategies.

AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression (2023.acl-long)

Copied to clipboard

Siyue Wu, Hongzhan Chen, Xiaojun Quan, Qifan Wang, Rui Wang

Prompting Test-Time Scaling Is A Strong LLM Reasoning Data Augmentation (2026.findings-acl)

Challenge:	Existing knowledge distillation methods focus on the transfer of model-specific knowledge but overlook data-specific information.
Approach:	They propose an attribution-driven knowledge distillation approach which explores the token-level rationale behind the teacher model and transfers attribution knowledge to the student model.
Outcome:	The proposed method outperforms state-of-the-art methods on the GLUE benchmark and shows that it is more efficient than existing methods.

Copied to clipboard

Sondos Mahmoud Bsharat, Zhiqiang Shen

BERT Learns to Teach: Knowledge Distillation with Meta Learning (2022.acl-long)

Challenge:	Large language models exhibit strong reasoning when guided by chain-of-thought exemplars . collecting large, high-quality reasoning datasets remains laborious and resource-intensive .
Approach:	They propose a prompt-space data augmentation framework for enhancing LLM reasoning . they use a pool of 90 randomly selected reasoning instances to elicit diverse reasoning trajectories .
Outcome:	The proposed framework improves accuracy over small-data benchmarks and generalization on out-of-domain reasoning evaluations.

Copied to clipboard

Wangchunshu Zhou, Canwen Xu, Julian McAuley

Autoregressive Knowledge Distillation through Imitation Learning (2020.emnlp-main)

Challenge:	Existing knowledge distillation methods are based on teacher model, but have drawbacks . a teacher model is fixed during training, but meta learning can improve student performance .
Approach:	They propose a meta learning framework that allows the teacher network to learn to better transfer knowledge to the student network.
Outcome:	Experiments show that MetaDistil can improve on existing methods and is less sensitive to student capacity and hyperparameters.

Copied to clipboard

Alexander Lin, Jeremy Wohlwend, Howard Chen, Tao Lei

Selective Knowledge Distillation for Neural Machine Translation (2021.acl-long)

Challenge:	Autoregressive models are ubiquitous in natural language processing due to the sequential nature of text generation.
Approach:	They propose a compression technique for autoregressive models driven by an imitation learning perspective on knowledge distillation.
Outcome:	The proposed method outperforms other distillation algorithms on translation and summarization tasks while increasing inference speed 14 times.

Copied to clipboard

Fusheng Wang, Jianhao Yan, Fandong Meng, Jie Zhou

Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judge (2025.findings-emnlp)

Challenge:	Neural Machine Translation models achieve state-of-the-art performance on many translation benchmarks.
Approach:	They propose a protocol that analyzes different impacts of samples by comparing various samples’ partitions.
Outcome:	The proposed methods yield up to +1.28 and +0.89 BLEU points improvements over the Transformer baseline, respectively.

Copied to clipboard

Zhuo Liu, Moxin Li, Xun Deng, Qifan Wang, Fuli Feng

Challenge:	LLM-as-a-Judge uses large language models to evaluate the quality of LLM generated responses, but training proxy judge models using evaluation data generated by powerful teacher models introduces a critical yet previously overlooked issue: teacher preference bias.
Approach:	They propose a new setting that incorporates an additional assistant model, which is not biased toward the teacher model’s responses, to complement the training data.
Outcome:	The proposed model reduces teacher preference bias while maintaining strong performance across six evaluation benchmarks.

Improving Stance Detection with Multi-Dataset Learning and Knowledge Distillation (2021.emnlp-main)

Copied to clipboard

Yingjie Li, Chenye Zhao, Cornelia Caragea

Enhancing Task-Specific Distillation in Small Data Regimes through Language Generation (2022.coling-1)

Challenge:	stance detection is a method to determine whether a text author is in favor of, against or neutral toward a specific target.
Approach:	They propose a method that applies instance-specific temperature scaling to the teacher and student predictions.
Outcome:	The proposed method outperforms the state-of-the-art on all datasets and on multiple datasets.

Copied to clipboard

Husam Quteineh, Spyridon Samothrakis, Richard Sutcliffe

Uncertainty-aware Parameter-Efficient Self-training for Semi-supervised Language Understanding (2023.findings-emnlp)

Challenge:	Large-scale pretrained language models have led to significant improvements in Natural Language Processing, but they come at the cost of high computational and storage requirements.
Approach:	They propose to distill knowledge from larger models to smaller ones through pseudo-labels on task-specific datasets.
Outcome:	The proposed approach improves on the SST-2, MRPC, YELP-2, and TREC-6 datasets.

Copied to clipboard

Jianing Wang, Qiushi Sun, Nuo Chen, Chengyu Wang, Jun Huang, Ming Gao, Xiang Li

Towards Better Entity Linking with Multi-View Enhanced Distillation (2023.acl-long)

Challenge:	Existing methods for pre-trained language models rely on noisy data, which can be expensive if all parameters are updated.
Approach:	They propose a self-training framework that incorporates Monte Carlo dropouts into the model and judiciously selects reliable pseudo-labeled examples based on confidence and certainty.
Outcome:	The proposed framework improves performance and efficiency over multiple tasks over multiple datasets.

Copied to clipboard

Yi Liu, Yuan Tian, Jianxun Lian, Xinlong Wang, Yanan Cao, Fang Fang, Wen Zhang, Haizhen Huang, Weiwei Deng, Qi Zhang

DistillCSE: Distilled Contrastive Learning for Sentence Embeddings (2023.findings-emnlp)

Challenge:	Entity linking is a fundamental task in Natural Language Processing (NLP), connecting mentions within unstructured contexts to their corresponding entities in a Knowledge Base (KB).
Approach:	They propose a dual-encoder framework that can efficiently match mentions to two-encoding frameworks by a global-view.
Outcome:	The proposed framework achieves state-of-the-art on several entity linking benchmarks.

Copied to clipboard

Jiahao Xu, Wei Shao, Lihui Chen, Lemao Liu

Mock Worlds, Real Skills: Building Small Agentic Language Models with Synthetic Tasks, Simulated Environments, and Rubric-Based Rewards (2026.acl-long)

Challenge:	Existing approaches to sentence embeddings are based on contrastive learning (CL) .
Approach:	They propose a framework which performs contrastive learning under the self-training paradigm with knowledge distillation and propose 'Group-P shuffling strategy' and averaging logits from multiple teacher components.
Outcome:	The proposed framework outperforms many strong baseline methods and yields a new state-of-the-art performance.

Copied to clipboard

Yuanjie Lyu, Chengyu Wang, Lei Shen, Jun Huang, Tong Xu

Revisiting Knowledge Distillation for Autoregressive Language Models (2024.acl-long)

Challenge:	Existing agentic training data are narrow in task variety and easily solved . real-world APIs lack diversity and are unstable for large-scale reinforcement learning rollout processes.
Approach:	They propose a framework that synthesizes diverse tool-use training data and simulates complete environments.
Outcome:	The proposed framework synthesizes diverse tool-use training data and simulates complete environments.

Copied to clipboard

Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, Dacheng Tao

RASPberry: Retrieval-Augmented Monte Carlo Tree Self-Play with Reasoning Consistency for Multi-Hop Question Answering (2025.findings-acl)

Challenge:	Autoregressive language models (LMs) are expensive and memory intensive, preventing the development of industrial applications.
Approach:	They propose an adaptive teaching approach to improve the KD of autoregressive language models by distilling knowledge into a small student model.
Outcome:	The proposed method can achieve consistent and significant performance gains across all model types and sizes.

Copied to clipboard

Baixuan Li, Yunlong Fan, Tianyi Ma, Miao Gao, Chuanqi Shi, Zhiqiang Gao

Accelerating Multiple Intent Detection and Slot Filling via Targeted Knowledge Distillation (2023.findings-emnlp)

Challenge:	Existing methods for generating and analyzing multiple document knowledge are not effective for multi-hop question answering.
Approach:	They propose a Monte Carlo tree-based approach to inference-time scaling using RASPberry.
Outcome:	Experimental results show that the proposed method achieves better inference-time scaling on smaller LLMs.

Copied to clipboard

Xuxin Cheng, Zhihong Zhu, Wanshi Xu, Yaowei Li, Hongxiang Li, Yuexian Zou

Distilled Dual-Encoder Model for Vision-Language Understanding (2022.emnlp-main)

Challenge:	Existing non-autoregressive Spoken Language Understanding models suffer from multi-modality problem . current methods have little prior knowledge about the reference during inference .
Approach:	They propose a Targeted Knowledge Distillation Framework (TKDF) for multi-intent SLU that utilizes the knowledge distillation method to improve the performance.
Outcome:	The proposed model outperforms existing models on two public multi-intent datasets while speeding up by over 4.5 times.

Copied to clipboard

Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, Furu Wei

Self-chats from Large Language Models Make Small Emotional Support Chatbot Better (2024.acl-long)

Challenge:	Experimental results show that the proposed cross-modal attention distillation is crucial to the success of our framework.
Approach:	They propose a framework that distills knowledge of fusion-encoder teacher into dual-encoding student model.
Outcome:	The proposed model is competitive with the fusion-encoder teacher model in performance, but suffers from a lack of deep cross-modal interactions.

Copied to clipboard

Zhonghua Zheng, Lizi Liao, Yang Deng, Libo Qin, Liqiang Nie

One-Teacher and Multiple-Student Knowledge Distillation on Sentiment Classification (2022.coling-1)

Challenge:	Large Language Models (LLMs) have shown strong generalization abilities to excel in various tasks, including emotion support conversations.
Approach:	They propose an iterative expansion framework to prompt large teacher model to curate an expansive emotion support dialogue dataset.
Outcome:	The proposed model outperforms the teacher model in some cases . the proposed model is based on an iterative expansion framework and is available on github.com/pandazzh2020/ExTES.

Copied to clipboard

Xiaoqin Chang, Sophia Yat Mei Lee, Suyang Zhu, Shoushan Li, Guodong Zhou

Fine-tuning Smaller Language Models for Question Answering over Financial Documents (2024.findings-emnlp)

Challenge:	Existing knowledge distillation models require large computing resources and long inference time to perform.
Approach:	They propose a one-teacher and multiple-student knowledge distillation approach to distill a deep pre-trained teacher model into multiple shallow student models with ensemble learning.
Outcome:	The proposed method achieves better results with fewer parameters and extremely high speedup ratios on three sentiment classification tasks.

Copied to clipboard

Karmvir Phogat, Sai Akhil Puranam, Sridhar Dasaratha, Chetan Harsha, Shashishekar Ramakrishna

Question Answering

Challenge:	Recent research has shown that smaller language models can acquire substantial reasoning abilities when fine-tuned with reasoning exemplars crafted by a significantly larger teacher model.
Approach:	They propose to fine-tune several smaller model to generate programs that encode the required financial reasoning and calculations.
Outcome:	The proposed model outperforms the teacher model in the financial domain by adjusting the entity extraction for the specific data format.

Efficiently Selecting Response Generation Strategies for Synthetic Data Construction by Self-Aligned Perplexity (2025.findings-emnlp)

Copied to clipboard

Xuan Ren, Qi Chen, Lingqiao Liu

Text Generation

Challenge:	Using a small sample of data, we find that perplexity is suboptimal in characterizing “familiarity” .
Approach:	They propose a method that assesses a small subset of generated data to estimate suitability for a specific target LLM.
Outcome:	The proposed method assesses a small subset of generated data to estimate suitability for a specific target LLM.

Towards Adapting Open-Source Large Language Models for Expert-Level Clinical Note Generation (2025.findings-acl)

Copied to clipboard

Hanyin Wang, Chufan Gao, Bolun Liu, Qiping Xu, Guleid Hussein, Mohamad El Labban, Kingsley Iheasirim, Hariprasad Reddy Korsapati, Chuck Outcalt, Jimeng Sun

Distillation Enhanced Generative Retrieval (2024.findings-acl)

Challenge:	Proprietary Large Language Models (LLMs) have demonstrated promising capabilities in clinical text summarization tasks.
Approach:	They propose a domain- and task-specific adaptation process for an open-source LLaMA-2 model . LLama-2 can generate high-quality clinical notes from outpatient patient-doctor dialogues .
Outcome:	The proposed model can generate clinical notes comparable to those authored by physicians.

Copied to clipboard

Yongqi Li, Zhen Zhang, Wenjie Wang, Liqiang Nie, Wenjie Li, Tat-Seng Chua

Hard Gate Knowledge Distillation - Leverage Calibration for Robust and Reliable Language Model (2022.emnlp-main)

Challenge:	Generative retrieval is a promising new paradigm in text retrieval that generates identifier strings of relevant passages as the retrieval target.
Approach:	They propose a framework that leverages generative language models to enhance generative retrieval by distillation.
Outcome:	The proposed framework achieves state-of-the-art performance among the generative retrieval methods.

Copied to clipboard

Dongkyu Lee, Zhiliang Tian, Yingxiu Zhao, Ka Chun Cheung, Nevin Zhang

To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation (2024.acl-long)

Challenge:	Existing knowledge distillation schemes focus on a teacher as a source of knowledge and a gauge to detect miscalibration of a student.
Approach:	They propose a method that uses a teacher model as a source of knowledge and a model as an error detector to detect miscalibration of a student.
Outcome:	The proposed scheme improves model generalization and significantly lowers calibration error.

Copied to clipboard

Abdul Waheed, Karima Kadaoui, Muhammad Abdul-Mageed

Self-Evolution Knowledge Distillation for LLM-based Machine Translation (2025.coling-main)

Challenge:	Existing models for multilingual automatic speech recognition (ASR) are computationallyintensive and lack proper comprehensive evaluations.
Approach:	They propose to distill knowledge from large teacher models into smaller student variants that are more efficient.
Outcome:	The proposed model outperforms existing models on standard benchmarks and dialectal data.

Copied to clipboard

Yuncheng Song, Liang Ding, Changtong Zan, Shujian Huang

A Study on Knowledge Distillation from Weak Teacher for Scaling Up Pre-trained Language Models (2023.findings-acl)

Challenge:	Existing knowledge distillation strategies for large language models minimize output distributions between student and teacher models indiscriminately for each token.
Approach:	They propose a distillation strategy that integrates teacher and one-hot distribution of ground truth into the student distribution as prior knowledge, which promotes the distillation process.
Outcome:	The proposed method brings an average improvement of approximately 1.4 SacreBLEU points across four translation directions in the WMT22 test sets.

Copied to clipboard

Hayeon Lee, Rui Hou, Jongpil Kim, Davis Liang, Sung Ju Hwang, Alexander Min

Process-Supervised Reinforcement Learning for Code Generation (2025.emnlp-main)

Challenge:	a study shows that DWT can be effective in the vision domain and natural language processing pre-training stages.
Approach:	They examine three key factors to optimize Distillation from Weak Teacher (DWT) DWT is a method of transferring knowledge from a weaker teacher model to a larger student model to improve its performance.
Outcome:	a new study examines three key factors to optimize DWT in NLP pre-training scenarios . the impact of teacher model quality and guidelines for adjusting the weighting value for DW T loss are examined .

Copied to clipboard

Yufan Ye, Ting Zhang, Wenbin Jiang, Hua Huang

HuatuoGPT, Towards Taming Language Model to Be a Doctor (2023.findings-emnlp)

Challenge:	Existing reinforcement learning strategies based on outcome supervision have shown effectiveness in code generation tasks, but their effectiveness in the field of code generation remains limited.
Approach:	They propose a method that uses a teacher model to mutate and refactor statements and a compiler to automatically label them.
Outcome:	The proposed method improves performance in complex code generation tasks.

Copied to clipboard

Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Guiming Chen, Jianquan Li, Xiangbo Wu, Zhang Zhiyi, Qingying Xiao, Xiang Wan, Benyou Wang, Haizhou Li

Case2Code: Scalable Synthetic Data for Code Generation (2025.coling-main)

Challenge:	Experimental results show that the distilled language model outperforms its teacher model (ChatGPT) in most cases.
Approach:	They propose a Large Language Model (LLM) that leverages both distilled data from ChatGPT and real-world data fromdoctors in the supervised fine-tuning stage.
Outcome:	The proposed model outperforms the teacher model in most cases by using additional real-world data and RLMF to align the language model with the merits of both sources.

Copied to clipboard

Yunfan Shao, Linyang Li, Yichuan Ma, Peiji Li, Demin Song, Qinyuan Cheng, Shimin Li, Xiaonan Li, Pengyu Wang, Qipeng Guo, Hang Yan, Xipeng Qiu, Xuanjing Huang, Dahua Lin

MLAS-LoRA: Language-Aware Parameters Detection and LoRA-Based Knowledge Transfer for Multilingual Machine Translation (2025.acl-long)

Challenge:	Large Language Models (LLMs) have shown outstanding breakthroughs in code generation.
Approach:	They propose a case-to-code induction task that exploits the expressiveness and correctness of programs by incorporating LLMs into their training.
Outcome:	The proposed task improves distribution case-to-code induction and various coding generation tasks.

Copied to clipboard

Tianyu Dong, Bo Li, Jinsong Liu, Shaolin Zhu, Deyi Xiong

Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning (2025.findings-acl)

Challenge:	Large language models (LLMs) have demonstrated strong performance even with limited parallel data.
Approach:	They propose a multiple language-aware LoRA knowledge transfer framework that selectively adapts LLMs to MT by transferring knowledge from a large teacher to a small student model.
Outcome:	The proposed framework outperforms baseline models on multilingual language pairs by +1.7 BLEU on average.

Copied to clipboard

Xinghao Chen, Zhijing Sun, Guo Wenjin, Miaoran Zhang, Yanjun Chen, Yirong Sun, Hui Su, Yijie Pan, Dietrich Klakow, Wenjie Li, Xiaoyu Shen

Boosting Zero-Shot Crosslingual Performance using LLM-Based Augmentations with Effective Data Selection (2024.findings-acl)

Challenge:	Large Language Models (LLMs) excel in reasoning tasks through Chain-of-Thought prompting.
Approach:	They examine the factors influencing CoT distillation including granularity, format and teacher model.
Outcome:	The proposed model is based on four teacher models and seven student models across seven mathematical and commonsense reasoning datasets.

Copied to clipboard

Barah Fazili, Ashish Agrawal, Preethi Jyothi

Low-Resource NLP

Challenge:	Large language models generate task-specific data via zero-shot prompting and promote cross-lingual transfer for low-resource target languages.
Approach:	They propose to use a teacher model to label LLM generations and use their label probabilities to identify a representative subset of diverse generations that boost zero-shot accuracies while being efficient.
Outcome:	The proposed models generate task-specific data via zero-shot prompting and promote cross-lingual transfer for low-resource target languages.

daDPO: Distribution-Aware DPO for Distilling Conversational Abilities (2025.findings-acl)

Copied to clipboard

Zhengze Zhang, Shiqi Wang, Yiqun Shen, Simin Guo, Dahua Lin, Xiaoliang Wang, Nguyen Cam-Tu, Fei Tan

Boosting Multi-modal Keyphrase Prediction with Dynamic Chain-of-Thought in Vision-Language Models (2025.emnlp-main)

Challenge:	Knowledge distillation (KD) with Direct Preference Optimization (DPO) has emerged as a promising approach to enhance the conversational abilities of smaller models using a larger teacher model.
Approach:	They propose a framework that integrates the teacher's distributional information into DPO distillation while preserving theoretical guarantees.
Outcome:	The proposed framework outperforms existing methods in restoring performance for pruned models and enhancing smaller models within the same LLM family.

Copied to clipboard

Qihang Ma, Shengyu Li, Jie Tang, Dingkang Yang, null Chenshaodong, Yingyi Zhang, Chao Feng, Ran Jiao

A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training (2023.acl-long)

Challenge:	Multi-modal keyphrase prediction (MMKP) aims to produce concise, informative phrases that capture the essence of cross-modal inputs.
Approach:	They propose to use vision-language models to generate conclusive phrases using multiple modalities of input information.
Outcome:	The proposed methods outperform existing methods on absence and unseen scenarios and overestimate model capability due to overlap in training tests.

Copied to clipboard

Nitay Calderon, Subhabrata Mukherjee, Roi Reichart, Amir Kantor

Large Language Models Are Reasoning Teachers (2023.acl-long)

Challenge:	Modern Natural Language Generation models come with massive computational and storage requirements.
Approach:	They propose a method that applies word-level knowledge distillation to multiple PTs generated by both teacher and student.
Outcome:	The proposed techniques can be used to compress natural language models while preserving their performance.

Copied to clipboard

Namgyu Ho, Laura Schmid, Se-Young Yun

Beyond Semantic Similarity: Appraisal-Guided Chain-of-Thought Reasoning and Retrieval for Multimodal Emotional Support Conversations (2026.findings-acl)

Challenge:	Recent studies have shown that chain-of-thought (CoT) prompting can elicit language models to solve complex reasoning tasks step-by-step.
Approach:	They propose a method that uses large model samples as reasoning teachers to fine-tune smaller models.
Outcome:	The proposed method outperforms prompt-based methods and the teacher model in many tasks and extends it by leveraging the teacher's ability to generate multiple rationales for each original sample.

Copied to clipboard

Yuqi Chu, Lizi Liao, Jinggui Liang, Boyang Li, Richang Hong

EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning (2023.findings-acl)

Challenge:	Existing retrieval-augmented generation paradigms rely on semantic similarity to retrieve historical dialogues that are surface analogous but therapeutically incongruent.
Approach:	They propose to use appraisal-guided reasoning chains to generate appraisal-based reasoning chains and apply a dual-signal verification mechanism to verify and correct them.
Outcome:	Extensive experiments on two ESC benchmarks show that the proposed model significantly outperforms state-of-the-art models.

Copied to clipboard

Tiannan Wang, Wangchunshu Zhou, Yan Zeng, Xinsong Zhang

Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages (2023.findings-acl)

Challenge:	Pre-trained vision-language models have achieved impressive results in a range of vision-linguistic tasks.
Approach:	They propose a distilling then pruning framework to compress large vision-language models into smaller, faster ones.
Outcome:	The proposed framework reduces the size of a pre-trained large vision-language model and improves its performance on vision-linguistic tasks.

Copied to clipboard

Shivanshu Gupta, Yoshitomo Matsubara, Ankit Chadha, Alessandro Moschitti

Mitigating Shortcuts in Language Models with Soft Label Encoding (2024.lrec-main)

Challenge:	Cross-Lingual Knowledge Distillation (CLKD) is a method to train AS2 models for low-resource languages without labeled data.
Approach:	They propose a method to train AS2 models for low-resource languages without labeled data . they use a translation-based WikiQA dataset and a multilingual AS2 dataset .
Outcome:	The proposed method outperforms or rivals fine-tuning with labeled data and machine translation and the teacher model.

Copied to clipboard

Zirui He, Huiqi Deng, Haiyan Zhao, Ninghao Liu, Mengnan Du

Error-driven Data-efficient Large Multimodal Model Tuning (2025.acl-long)

Challenge:	Recent studies have shown that large language models rely on spurious correlations in the data for natural language understanding (NLU) tasks.
Approach:	They propose a framework for debiasing shortcuts and a dummy class to encode shortcuts into a model and use it to generate soft labels.
Outcome:	The proposed framework significantly improves out-of-distribution generalization while maintaining satisfactory in-district accuracy.

Copied to clipboard

Barry Menglong Yao, Qifan Wang, Lifu Huang

MoDE-CoTD: Chain-of-Thought Distillation for Complex Reasoning Tasks with Mixture of Decoupled LoRA-Experts (2024.lrec-main)

Challenge:	Large Multimodal Models (LMMs) have demonstrated impressive performance across numerous academic benchmarks, but task-specific tuning samples are often not readily available or expensive and time-consuming to obtain.
Approach:	They propose an error-driven data-efficient tuning framework that aims to efficiently adapt generic LMMs to newly emerging tasks without extensive task-specific training samples.
Outcome:	The proposed model achieves an average performance boost of 7.01% on seven tasks across three training data scales and three different training datascales.

Copied to clipboard

Xiang Li, Shizhu He, Jiayu Wu, Zhao Yang, Yao Xu, Yang jun Jun, Haifeng Liu, Kang Liu, Jun Zhao

Dual-Space Knowledge Distillation for Large Language Models (2024.emnlp-main)

Challenge:	Current Chain-of-thought Distillation methods hinder CoT reasoning performance . student models are separately distilled from specific reasoning tasks . parameter update of student models severely harms CoT ability on unseen reasoning tasks.
Approach:	They propose a method which distills Chain-of-thought reasoning ability of large language models to much smaller student models.
Outcome:	The proposed method improves the reasoning ability of large language models on 14 datasets.

Copied to clipboard

Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, Jinan Xu

The Emperor’s New Reasoning: Format Imitation Overshadows Genuine Mathematical Understanding in SFT (2025.emnlp-main)

Challenge:	Existing large language models (LLMs) have strong generalization abilities due to their huge model capacities.
Approach:	They propose a dual-space knowledge distillation framework that unifies the output spaces of the two models for KD.
Outcome:	The proposed framework outperforms existing white-box KD frameworks on task-agnostic instruction-following benchmarks and can automatically align representations of two models with different vocabularies.

Copied to clipboard

Linyao Yang, Jian-Tao Huang, Yafei Lu, Zhenhui Jessie Li, Guirong Xue

Flipping Knowledge Distillation: Leveraging Small Models’ Expertise to Enhance LLMs in Text Matching (2025.acl-long)

Challenge:	Recent advances in large language models (LLMs) have yielded impressive gains on mathematical reasoning benchmarks via supervised fine-tuning (SFT).
Approach:	They investigate the mechanisms behind SFT improvements in small-scale large language models by examining four key questions: (1) Are performance gains primarily due to format alignment rather than reasoning? (2) Can high-quality supervision encourage genuine reasoning? (4) Are format alignment gains consistent across model sizes and architectures?
Outcome:	The proposed models outperform the proprietary models on OlympiadBench and Omni-Math, but lack the brittleness of the models under perturbations to test their reasoning abilities.

Copied to clipboard

Mingzhe Li, Jing Xiang, Qishen Zhang, Kaiyang Wan, Xiuying Chen

Model Calibration for Emotion Detection (2025.findings-emnlp)

Challenge:	Large Language Models (LLMs) have demonstrated remarkable capabilities in acquiring diverse knowledge, making them highly effective across a wide range of tasks.
Approach:	They propose a flipped knowledge distillation paradigm where LLM learns from SLM . they propose to reinterpret LLMs as encoder-decoder models using LoRA .
Outcome:	The proposed model has been deployed in an online application environment and validated on financial and healthcare benchmarks and real-world applications.

Copied to clipboard

Mihaela Petre-Vlad, Cornelia Caragea, Florentina Hristea

Beyond Overlap Metrics: Rewarding Reasoning and Preferences for Faithful Multi-Role Dialogue Summarization (2026.findings-acl)

Challenge:	a MixUp method is used to calibrate emotion detection models based on knowledge distillation and the MixUp data augmentation technique.
Approach:	They propose a method that uses knowledge distillation and the MixUp data augmentation technique to enhance the trustworthiness of emotion detection models.
Outcome:	The proposed method improves the accuracy of the teacher models and the student models.

Copied to clipboard

Xiaoyong Mei, Tingting Zuo, Da Chen, Guangyu Hu, Xiangyu Wen, Chao Duan, Mingyan Zhang, Fudan Zheng

Selective Knowledge Distillation: Fusing LLM Semantic Strengths with DNN Efficiency for Binary Code Similarity Detection (2026.acl-long)

Challenge:	Existing methods for multi-role dialogue summarization favor surface-level imitation of references rather than genuine gains in faithfulness or alignment with human preferences.
Approach:	They propose a framework that couples explicit cognitive-style reasoning with reward-based optimization for multi-role dialogue summarization.
Outcome:	The proposed framework matches strong baselines on ROUGE and BERTScore, while in-depth analysis on SAMSum shows clear gains in factual faithfulness and model-based preference alignment.

Copied to clipboard

Shize Zhou, Peiyu Liu, Lirong Fu, Tong Ye, Wenhai Wang

CIRAG: Construction–Integration Retrieval and Adaptive Generation for Multi-hop Question Answering (2026.acl-long)

Challenge:	BinSKD is a binary code similarity detection technique that can be used in bug detection, patch analysis, and malware detection.
Approach:	They propose to leverage an LLM-based BCSD method as the teacher model and transfer its knowledge of high-level program semantics to various DNN-based student models.
Outcome:	The proposed method yields Recall@1 improvements of 14.5%–91.2% for DNN-based BCSD methods and enables HermesSim to match the teacher’s performance with orders-of-magnitude efficiency.

Copied to clipboard

Zili Wei, Yilin Wang, Xiaocui Yang, Shi Feng, Weidong Bao, Daling Wang, Zihan Wang, Yifei Zhang

Information Retrieval

Challenge:	Existing methods for iterative retrieval-augmented generation (iRAG) suffer from greedy single-path expansion and granularity–demand mismatch .
Approach:	They propose a model that constructs candidate triples and history-conditionally integrates them to distill core triples to generate the next-hop query.
Outcome:	The proposed model mitigates the greedy single-path expansion and granularity–demand mismatch by preserving multiple plausible evidence chains.

Self-Knowledge Distillation for Knowledge Graph Embedding (2024.lrec-main)

Copied to clipboard

Haotian Xu, Yuhua Wang, Jiahui Fan

Temporal reasoning for timeline summarisation in social media (2025.acl-long)

Challenge:	Knowledge graph embedding (KGE) is an important task for many downstream applications.
Approach:	They propose to use self-knowledge distillation to learn a low-dimensional model from a pre-trained high-dimensional one.
Outcome:	The proposed model can improve model performance while maintaining lightweight structure.

Copied to clipboard

Jiayu Song, Mahmud Elahi Akhter, Dana Atzil-Slonim, Maria Liakata

Cognitive-Uncertainty Guided Knowledge Distillation for Accurate Classification of Student Misconceptions (2026.findings-acl)

Challenge:	Existing temporal reasoning datasets focus on pair-wise event relationships.
Approach:	They propose a temporal reasoning dataset focused on temporal relationships among sequential events within narratives that combines temporal thinking with timeline summarisation through a knowledge distillation framework.
Outcome:	The proposed model achieves superior performance on mental health-related timeline summarisation tasks, highlighting the importance and generalisability of leveraging temporal reasoning to improve timeline summaries.

Copied to clipboard

Qirui Liu, Hao Chen, Weijie Shi, Jiajie Xu, Jia Zhu

Information Retrieval

Challenge:	Existing methods for identifying student misconceptions overlook students' reasoning processes, authors report .
Approach:	They propose a knowledge distillation framework that mines high-value samples from existing data.
Outcome:	The proposed framework outperforms sota LLM and standard fine-tuned 72B models on cross-topic tests.

StepER: Step-wise Knowledge Distillation for Enhancing Reasoning Ability in Multi-Step Retrieval-Augmented Language Models (2025.emnlp-main)

Copied to clipboard

Kyumin Lee, Minjin Jeon, Sanghwan Jang, Hwanjo Yu

Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translation (2025.acl-long)

Challenge:	Existing knowledge distillation methods overlook the need for different reasoning abilities at different steps, hindering transfer in multi-step retrieval-augmented frameworks.
Approach:	They propose a method that uses step-wise supervision to align with evolving information and reasoning demands across stages.
Outcome:	The proposed method outperforms previous methods on multi-hop QA benchmarks with an 8B model achieving performance comparable to a 70B teacher model.

Copied to clipboard

Fan Yin, Zifeng Wang, I-Hung Hsu, Jun Yan, Ke Jiang, Yanfei Chen, Jindong Gu, Long Le, Kai-Wei Chang, Chen-Yu Lee, Hamid Palangi, Tomas Pfister

ThinkTuning: Instilling Cognitive Reflections without Distillation (2025.emnlp-main)

Challenge:	Large language models have been shown to be effective in multi-turn interactions . however, their performance may be limited in complex, multi-turned interactions involving users and multiple tools.
Approach:	They propose a framework for synthesizing high-quality training trajectories to enhance the function calling capability of large language model agents in multi-turn conversations with humans.
Outcome:	The proposed model outperforms the teacher model by 68.01 on BFCL-v3 and 73.30 on ToolQuery.

Copied to clipboard

Aswin Rrv, Jacob Dineen, Divij Handa, Md Nayem Uddin, Mihir Parmar, Chitta Baral, Ben Zhou

Lizard: An Efficient Linearization Framework for Large Language Models (2026.acl-long)

Challenge:	Recent advances in test-time scaling have led to the emergence of thinking LLMs that exhibit self-reflective behaviors and multi-step reasoning.
Approach:	They propose a GRPO-based interactive training approach that augments the rollouts of a student model with the guidance of . a teacher poses a problem, lets the student try an answer, then gives corrective feedback–enough to point the mind in the right direction and then show the correct solution.
Outcome:	The proposed method shows 3.69% improvement over zero-shot baselines and 2.08% and 3.99% improvement over the vanilla-GRPO baselines.

Copied to clipboard

Chien Van Nguyen, Huy Huu Nguyen, Ruiyi Zhang, Hanieh Deilamsalehy, Puneet Mathur, Viet Dac Lai, Haoliang Wang, Jayakumar Subramanian, Ryan A. Rossi, Trung Bui, Nikos Vlassis, Franck Dernoncourt, Thien Huu Nguyen

MARD: Module-Aware Reasoning Distillation for Language Models with Adaptive Supervision (2026.acl-long)

Challenge:	Existing linearization frameworks that rely on softmax attention with quadratic time and memory complexity pose significant computational and memory bottlenecks for long-context applications.
Approach:	They propose a linearization framework that transforms pretrained Transformer-based Large Language Models into subquadratic architectures that closely approximate softmax attention while preserving model quality.
Outcome:	Experiments show that the proposed framework outperforms existing methods by 9.4 - 24.5 points on the 5-shot MMLU benchmark and demonstrating superior associative recall.

Copied to clipboard

Wenqi Yang, Jianjun Li, Zhibo Zhang, Mingqian Ding, Yushen Fang

Can Compact Language Models Search Like Agents? Distillation-Guided Policy Optimization for Preserving Agentic RAG Capabilities (2026.acl-long)

Challenge:	Multi-step reasoning remains challenging for language models with limited capacity . et al., 2025) demonstrate remarkable reasoning capabilities across diverse tasks .
Approach:	They propose a module-aware reasoning distillation framework that explicitly targets key Transformer components for effective reasoning transfer.
Outcome:	The proposed framework targets key components for effective reasoning transfer . it adopts an offline distillation setting, where a strong teacher model provides reasoning trajectories in advance .

Copied to clipboard

Rikuto Kotoge, Mai Nishimura, Jiaxin Ma

Long-Context Reasoning Through Proxy-Based Chain-of-Thought Tuning (2026.acl-long)

Challenge:	Existing agentic RAG systems rely on large language models with billions of parameters.
Approach:	They propose a method to elicit agentic RAG behaviors from compact models . they propose ARC, which uses cold-start initialization and teacher guidance .
Outcome:	The proposed method outperforms the larger teacher model in some cases.

Copied to clipboard

Miao Li, Irina Saparina, Alexander Gurung, Mirella Lapata

LEAF: Knowledge Distillation of Text Embedding Models with Teacher-Aligned Representations (2026.acl-long)

Challenge:	Recent large language models support inputs of up to 10 million tokens, yet they perform poorly on long-context tasks.
Approach:	They propose a training framework that transfers reasoning capabilities from proxy contexts to full long contexts.
Outcome:	The proposed framework outperforms baseline models with reduced computational overhead.

Copied to clipboard

Robin Vujanic, Thomas Rückstieß